Quality scoring system for advertisements and content in an online system

ABSTRACT

A server system of an online information system displays advertising items and content items retrieved from storage devices as a stream viewable by a user on a user device. The advertisement items and the content items are ordered in the stream by a ranking score computed for each of the advertisement items and each of the content items. A quality scoring system determines an affinity score between a user and a present content item based on features of the present content item matching user profile parameters associated with the user and identifies post-interaction satisfaction with a prior content item. The quality scoring system determines a quality score based on the affinity score and the post-interaction satisfaction. The quality score is used for ordering items in the stream. The server system transmits a web page including the stream to a user device over a network. In this manner, advertising items and content items compete in a unified marketplace for inclusion in the stream for viewing by the end user.

PRIORITY

This application claims priority as a continuation application to U.S.application Ser. No. 13/932,655, filed on Jul. 1, 2013, now U.S. Pat.No. 9,460,451, issued on Oct. 4, 2016, entitled “QUALITY SCORING SYSTEMFOR ADVERTISEMENTS AND CONTENT IN AN ONLINE SYSTEM” the entiredisclosure of which is hereby incorporated by reference.

BACKGROUND

This application relates generally to data processing systems. Moreparticularly, this application relates to systems and methods fordisplaying revenue-generating information such as advertisements andnon-revenue-generating information such as content together online.

Online advertising has become increasingly popular as a way foradvertisers to publicize information about goods and services topotential customers and clients. An advertiser can implement anadvertising campaign using internet-accessible facilities of onlineproviders such as Yahoo! Inc. The online provider serves to connect theadvertiser with users accessing online resources such as search enginesand news and information sites. Advertisements (“ads”) of the advertiserare provided to the users to inform and attract the attention of theusers.

Some online providers provide a stream of content and other informationon a web page. The web page may be accessed by users on devices such asdesktop computers, portable computers such as laptops and handhelddevices such as tablets and smartphones, or media devices such astelevisions. The stream is presented on the web page as a sequence ofitems displayed, one item after another, for example, down the web pagewhen viewed on the display of a device. In some cases, the stream may beupdated with new content at the top or bottom of the page upon certainevents, such as the elapse of a certain period of time, the scrolling ofa mouse, or the click of a spacebar.

Advertising items, also referred to herein as “stream ads,” are insertedinto the stream of content, supplementing the sequence of items. Streamads may be formatted to visually match the surrounding stream of contentso as to appear native to the stream. Alternatively, stream ads may beformatted to complement the surrounding stream of content so as to bemore eye-catching.

Streams are becoming common in online presentation in part because theyprovide added flexibility for web site designers and advertisers. If astream is not used to present data on a web page, the web page must havepre-defined sections. Only certain types of information, havingspecified sizes, shapes or content, can be presented in the pre-definedsections. A stream allows any number and size and shape of content to beincluded. A stream also lowers the cognitive load on the viewer whenprocessing information associated with different items of content oradvertisements by removing the cognitive overhead associated withswitching to a different visual format or perspective.

It is desirable to manage the flow of content and advertisements in astream in order to, in turn, manage the experience of users andadvertisers who interact with the online provider. Further limitationsand disadvantages of conventional and traditional approaches will becomeapparent to one of skill in the art, through comparison of such systemswith some aspects of the disclosure set forth in the remainder of thepresent application with reference to the drawings.

BRIEF SUMMARY

In accordance with the systems, products and methods disclosed here, anonline provider may control the location, number and spatial andtemporal frequency of stream ads in a stream of content viewable by auser on a web page. The stream may be viewed as a unified marketplace inwhich both revenue-generating items and non-revenue-generating itemssuch as advertising items and content items, respectively, compete forpresentation in the stream. Scoring, ranking, and pricing techniquespermit commensurate treatment of all items (whether revenue generatingfor the online provider or not). Additional business rules for contentitems and advertising items provide further degrees of freedom for theonline provider in determining how content items and advertisement itemsare presented in a stream.

One aspect of scoring relates to content items that are to be presentedto a user. A quality score may be developed to reflect the relativequality or desirability of a content item to a user or a segment ofusers. The quality score for the content item may be used to determine abid amount for a content that may in turn be used to determine a rankingscore for the content item. For example, in one embodiment, a qualityscore for the content item is used as an index to a table which storesbid values for advertising items that are competing for the attention ofthe same users as the content item. The bids for the advertising itemsare used to automatically calculate a bid for a content item. Using thebid for the content item in combination with a measure described asclickability and a measure described as satisfaction, a ranking scorefor the content item can be derived.

In another embodiment, the quality score for the quality score is afunction of the popularity of the content item and the affinity betweenthe user and the content item. Popularity for a single content item canbe determined using the popularity of similar content items and a user'shistorical interests as recorded by user interaction with prior contentitems. Affinity may be determined by correlating a content item withparameters of the user's profile as specified by the user or based onthe user's past interaction with content items.

For the online provider, controlling the location, number and frequencyof ads in a stream can help manage the experience of users and ofadvertisers with the web site of the online provider. Providing too manyads can result in a less satisfying experience for a user. Providing toofew ads can reduce or eliminate advertiser involvement with the website. Selecting the most appropriate content items and advertisementitems for the user can keep the user engaged with the web site andensure the user will return to the web site. User engagement in turndrives the confidence and involvement advertisers who place stream adson the web site. The present disclosure generally describes a unifiedmarketplace in which every item of information presented by an onlineprovider, from revenue generating advertisements to paid content, isscored, priced according to explicit or implicit bids, and ranked forpresentation in a unitary format, such as a stream.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an example online information system;

FIG. 2 is an example illustrating a display ad as modified for displayin a stream display;

FIG. 3 is a flow diagram illustrating one embodiment a method forranking and displaying a stream of advertising items and content itemsin an online information system; and

FIG. 4 is an example process for displaying content in a streaming mediafeed according to a quality score computed using clickability andpost-click satisfaction scores.

DETAILED DESCRIPTION

Subject matter will now be described more fully hereinafter withreference to the accompanying drawings, which form a part hereof, andwhich show, by way of illustration, specific exemplary embodiments.Subject matter may, however, be embodied in a variety of different formsand, therefore, covered or claimed subject matter is intended to beconstrued as not being limited to any example embodiments set forthherein; example embodiments are provided merely to be illustrative.Likewise, a reasonably broad scope for claimed or covered subject matteris intended. Among other things, for example, subject matter may beembodied as methods, devices, components, or systems. The followingdetailed description is, therefore, not intended to be limiting on thescope of what is claimed.

Throughout the specification and claims, terms may have nuanced meaningssuggested or implied in context beyond an explicitly stated meaning.Likewise, the phrase “in one embodiment” as used herein does notnecessarily refer to the same embodiment and the phrase “in anotherembodiment” as used herein does not necessarily refer to a differentembodiment. It is intended, for example, that claimed subject matterincludes combinations of example embodiments in whole or in part.

In general, terminology may be understood at least in part from usage incontext. For example, terms, such as “and”, “or”, or “and/or,” as usedherein may include a variety of meanings that may depend at least inpart upon the context in which such terms are used. Typically, “or” ifused to associate a list, such as A, B or C, is intended to mean A, B,and C, here used in the inclusive sense, as well as A, B or C, here usedin the exclusive sense. In addition, the term “one or more” as usedherein, depending at least in part upon context, may be used to describeany feature, structure, or characteristic in a singular sense or may beused to describe combinations of features, structures or characteristicsin a plural sense. Similarly, terms, such as “a,” “an,” or “the,” again,may be understood to convey a singular usage or to convey a pluralusage, depending at least in part upon context. In addition, the term“based on” may be understood as not necessarily intended to convey anexclusive set of factors and may, instead, allow for existence ofadditional factors not necessarily expressly described, again, dependingat least in part on context.

An online information system places advertisements of advertisers withincontent services made available to end users, such as web pages, mobileapplications (“apps”), TV apps, or other audio or visual contentservices. The advertisements are provided along with other content. Theother content may include any combination of text, graphics, audio,video, or links to such content. The advertisements are conventionallyselected based on a variety of criteria including those specified by theadvertiser. The advertiser conventionally defines an advertisingcampaign to control how and when advertisements are made available tousers and to specify the content of those advertisements.

Streams are becoming common in online presentation because they provideflexibility for content providers who source content items for thestream, advertisers who source advertising items for the stream and forthe online provider who combines the content items and the advertisingitems to produce the stream. A stream allows any number and size andshape of content items and advertising items to be included in thestream. The elements of the stream may be sorted by relevance or by anysuitable parameter. A stream also lowers the cognitive load on theviewer when processing when processing information associated withdifferent items of content or advertisements by removing the cognitiveoverhead associated with switching to a different visual format orperspective.

The stream may be viewed as a unified marketplace where content itemsand advertising items compete for placement or inclusion in the stream.The participants in the marketplace are advertisers who sponsor orprovide the advertising items and content providers who sponsor orprovide the content items. The stream and the marketplace may be hostedor managed by an online provider such as Yahoo! Inc. The online providermay also provide ads for its own products and services or its owncontent items to the stream.

Advertisers interact with equipment of the online provider to create orprovide online advertisements. The online advertisements includeadvertising content stored in a database or other memory in associationwith identification of the advertiser and one or more bid amounts. Theadvertising content may include text or graphics or both and a link to alanding page to which the user's browser is redirected upon clicking thelink. The bid amount represents an amount of money the advertiser willpay upon an event pertaining to the advertisement. The event may be animpression or viewing of the advertisement by a user, a click through orother selection of the advertisement by the user viewing theadvertisement, or an action following viewing the advertisement such asproviding credit card information or an email address. The bid amountmay be used for determining position of the advertisement in the streamin a manner to be described below. The online advertisement may includeother data as well including data defining how the advertisement willappear in the stream.

The content items include information on a topic that may be of interestto a user. This information may include a link to another web pageproviding more information about the topic and a summary of informationabout the topic. In some embodiments, a content provider will associatea bid amount with a content item. Similar to bid amounts foradvertisements, the bid amount for a content item may be based on animpression, a click through, or another action. Also, the bid amount maybe used for determining position of the content item in the stream in amanner to be described below. Alternatively, a software based biddingagent may be employed to automatically bid on behalf of content items.

The content items and advertisement items are in competition forinclusion in the stream. The competition for slots in the stream may becleared using a Generalized Second Price (GSP) auction mechanism. In aGSP auction, the highest bidder gets the first slot, the second highestbidder gets the second slot and so. However, the highest bidder thenpays the price bid by the second highest bidder. This is similar to asponsored search marketplace although the bids in sponsored search areexpressed differently and the competition in a sponsored searchmarketplace is only between advertisements.

In one embodiment, an advertiser provides targeting predicates, an adsnippet and a bid. In some embodiments, the advertiser can provide abudget across multiple triples, referred to as targeting triples.Targeting predicates may be based on any type of market segment ofinterest to advertisers, including in one example, demographic markets,market segments based gender or age, behavioral segments based on userprofile information, or geographic markets. The bids may be cost perclick (CPC) bids, cost per impression (CPM) bids or cost per action(CPA) bids. The online provider may choose not to support all bid typesin all marketplaces.

What advertisers are allowed to bid for in large part determines theirbidding behavior. For the online provider who manages the unifiedmarketplace, there is a trade-off between allowing advertisers to bidfor very specific targets versus allowing advertisers to bid for morebroad targets.

The online provider may prefer markets that are thick with manycompeting advertisers to thin markets with few advertisers. The thickerthe market, the greater the potential for increased revenue to theonline provider. However, many advertisers are very interested inspecific types of user. These narrow-focused users will likely stay outof the marketplace unless they are allowed to bid more narrowly. Broadtargets lower the average value an advertiser derives since their adsmay be shown to users who may not be interested in their products. Lowerexpected values lead to lower bids.

Some of these trade-offs can be mitigated by pricing for performance, byusing excellent scoring algorithms and by preventing ads of lowrelevance from showing in the stream. Pricing for performance impliescharging only when a user responds to an advertisement. Advertiserswould prefer to pay only when users convert, such as by paying for aproduct or service. However, defining and tracking conversions andestimating conversions rates may be difficult to do reliably, somarketplace operators prefer charging by clicks which are more easilytracked and estimated. Charging per click can pose challenges. Forexample, not all clicks from users convert into sales for an advertiser.With too many clicks that do not result in a conversion, a low qualityscore for the ad may result.

Broad targeting requires precise scoring methods to maintain good userand advertiser experiences. Scoring is the process of assigning a valueto an advertisement or content item which value can then be used fordetermining which item should be included in the stream. This precisescoring may require that the online provider examine not just the adsnippet but also the contents of the landing page. In some embodiments,an advertisement may include additional information such as metadatathat is automatically collected or manually provided by the advertiserand used as signals to the scoring function.

Broad targeting may also add a difficulty in pricing for CPCadvertisements. In pricing the advertisement, it is important todistinguish between the quality of the match between a keyword and asearch term and the quality of the advertisement. The online operatormay choose to discount advertisers for poor quality matches, which arethe responsibility of the operator of the online marketplace which doesthe matching. The online operator may choose to charge a premium forpoor quality advertisements which are the responsibility of theadvertiser.

An exemplary system will now be described in which aspects of theunified marketplace for advertisement items and content items may beillustrated and described. Further details and optional embodiments willbe provided in connection with the drawings.

FIG. 1 is a block diagram of online information system 100. The onlineinformation system 100 in the exemplary embodiment of FIG. 1 includes anaccount server 102, and account database 104, a search engine 106, anadvertisement (ad) server 108, an ad database 110, a content database114, a content server 112 and a ranking engine 116. The onlineinformation system 100 may be accessible over a network 120 by one ormore advertiser devices such as advertiser device 122 and by one or moreuser devices such as user device 124. In various examples of such anonline information system, users may search for and obtain content fromsources over the network 120 or from the content database 114.Advertisers may provide advertisements for placement on web pages andother communications sent over the network to user devices such as theuser device 124. The online information system in one example isdeployed and operated by an online provider such as Yahoo! Inc.

The account server 102 stores account information for advertisers. Theaccount server 102 is in data communication with the account database104. Account information may include one or more database recordsassociated with each respective advertiser. Any suitable information maybe stored, maintained, updated and read from the account database 104 bythe account management server 102. Examples include advertiseridentification information, advertiser security information such aspasswords and other security credentials, and account balanceinformation.

The account server 102 may be implemented using any suitable device. Theaccount management server 102 may be implemented as a single server, aplurality of servers, or any other type of computing device known in theart. Preferably, access to the account server 102 is accomplishedthrough a firewall, not shown, which protects the account managementprograms and the account information from external tampering. Additionalsecurity may be provided via enhancements to the standard communicationsprotocols such as Secure HTTP or the Secure Sockets Layer.

The account server 102 may provide an advertiser front end to simplifythe process of accessing the account information of an advertiser. Theadvertiser front end may be a program, application or software routinethat forms a user interface. In one particular embodiment, theadvertiser front end is accessible as a web site with one or more webpages that an accessing advertiser may view on an advertiser device suchas advertiser device 122. The advertiser may view and edit account dataand advertisement data using the advertiser front end. After editing theadvertising data, the account data may then be saved to the accountdatabase 104.

The search engine 106 may be a computer system, one or more servers, orany other computing device known in the art. Alternatively, the searchengine 106 may be a computer program, instructions, or software codestored on a computer-readable storage medium that runs on a processor ofa single server, a plurality of servers, or any other type of computingdevice known in the art. The search engine 106 may be accessed, forexample, by user devices such as the user device 124 operated by a userover the network 120. The user device 124 communicates a user query tothe search engine 106. The search engine 106 locates matchinginformation using any suitable protocol or algorithm and returnsinformation to the user device 124. The search engine 106 may bedesigned to help users find information located on the Internet or anintranet. In a particular example, the search engine 106 may alsoprovide to the user device 124 over the network 120 a web page withcontent including search results, information matching the context of auser inquiry, links to other network destinations or information andfiles of information of interest to a user operating the user device124, as well as a stream of content items and advertisement itemsselected for display to the user.

The search engine 106 may enable a device, such as the user device 124or any other client device, to search for files of interest using asearch query. Typically, the search engine 106 may be accessed by aclient device via one or more servers or directly over the network 120.The search engine 106 may, for example, in one illustrative embodiment,comprise a crawler component, an indexer component, an index storagecomponent, a search component, a ranking component, a cache, a profilestorage component, a logon component, a profile builder, and one or moreapplication program interfaces (APIs). The search engine 106 may bedeployed in a distributed manner, such as via a set of distributedservers, for example. Components may be duplicated within a network,such as for redundancy or better access.

The ad server 108 operates to serve advertisements to user devices suchas the user device 124. Advertisements include data definingadvertisement information that may be of interest to a user of a userdevice. An advertisement may include text data, graphic data, imagedata, video data, or audio data. An advertisement may further includedata defining one or more links to other network resources providingsuch data. The other locations may be other locations on the internet,other locations on an intranet operated by the advertiser, or anyaccess.

For online information providers, advertisements may be displayed on webpages resulting from a user-defined search based at least in part uponone or more search terms. Advertising may be beneficial to users,advertisers or web portals if displayed advertisements are relevant tointerests of one or more users. Thus, a variety of techniques have beendeveloped to infer user interest, user intent or to subsequently targetrelevant advertising to users.

One approach to presenting targeted advertisements includes employingdemographic characteristics (e.g., age, income, sex, occupation, etc.)for predicting user behavior, such as by group. Advertisements may bepresented to users in a targeted audience based at least in part uponpredicted user behavior.

Another approach includes profile-type ad targeting. In this approach,user profiles specific to a user may be generated to model userbehavior, for example, by tracking a user's path through a web site ornetwork of sites, and compiling a profile based at least in part onpages or advertisements ultimately delivered. A correlation may beidentified, such as for user purchases, for example. An identifiedcorrelation may be used to target potential purchasers by targetingcontent or advertisements to particular users.

Yet another approach includes targeting based on content of a web pagerequested by a user. Advertisements may be placed on a web page or inassociation with other content that is related to the subject of theadvertisements. The relationship between the content and theadvertisement may be determined in any suitable manner. The overalltheme of a particular web page may be ascertained, for example, byanalyzing the content presented therein. Moreover, techniques have beendeveloped for displaying advertisements geared to the particular sectionof the article currently being viewed by the user. Accordingly, anadvertisement may be selected by matching keywords/and or phrases withinthe advertisement and the web page. One exemplary system and method aredisclosed in U.S. patent application Ser. No. 13/836,052, filed Mar. 15,2013, pending, entitled Efficient Matching of User Profiles withAudience Segments for Audience Buy. This application is incorporatedherein in its entirety by this reference.

The ad server 108 includes logic and data operative to format theadvertisement data for communication to the user device. The ad server108 is in data communication with the ad database 110. The ad database110 stores information including data defining advertisements to beserved to user devices. This advertisement data may be stored in the addatabase 110 by another data processing device or by an advertiser. Theadvertising data may include data defining advertisement creatives andbid amounts for respective advertisements.

For example, the advertising data may be formatted to an advertisingitem which may be included in a stream of content items and advertisingitems provided to a user device. The formatted advertising items arespecified by appearance, size, shape, text formatting, graphicsformatting and included information, which may all be standardized toprovide a consistent look for all advertising items in the stream. Atleast some advertising items may have an associated bid amount and maybe considered to be revenue generating items. The ad server 108 thenprovides the advertising items to other network devices such as theranking engine 116.

Further, the ad server 108 is in data communication with the network120. The ad server 108 communicates ad data and other information todevices over the network 120. This information may include advertisementdata communicated to a user device. This information may also includeadvertisement data and other information communicated with an advertiserdevice such as the advertiser device 122. An advertiser operating anadvertiser device may access the ad server 108 over the network toaccess information including advertisement data. This access may includedeveloping advertisement creatives, editing advertisement data, deletingadvertisement data, setting and adjusting bid amounts and otheractivities.

The ad server 108 may provide an advertiser front end to simplify theprocess of accessing the advertising data of an advertiser. Theadvertiser front end may be a program, application or software routinethat forms a user interface. In one particular embodiment, theadvertiser front end is accessible as a web site with one or more webpages that an accessing advertiser may view on the advertiser device.The advertiser may view and edit advertising data using the advertiserfront end. After editing the advertising data, the advertising data maythen be saved to the ad database 110 for subsequent communication inadvertisements to a user device.

The advertisement server 108 may be a computer system, one or moreservers, or any other computing device known in the art. Alternatively,the advertisement server 108 may be a computer program, instructionsand/or software code stored on a computer-readable storage medium thatruns on a processor of a single server, a plurality of servers, or anyother type of computing device known in the art.

The content server 112 is in data communication with the contentdatabase 114, the ad server 108 and the ranking engine 116. The contentserver 112 may access information about content items from either thecontent database 114 or from another location accessible over thenetwork 120. The content server 112 communicates data defining contentitems and other information to devices over the network 120. Thisinformation may include content data communicated to a user device. Thisinformation may also include content data and other informationcommunicated with a content provider operating a content providerdevice. A content provider operating a content provider device mayaccess the content server 112 over the network 120 to access informationincluding content data. This access may include developing contentitems, editing content items, deleting content items, setting andadjusting bid amounts and other activities.

The content server 112 may provide a content provider front end tosimplify the process of accessing the content data of a contentprovider. The content provider front end may be a program, applicationor software routine that forms a user interface. In one particularembodiment, the content provider front end is accessible as a web sitewith one or more web pages that an accessing content provider may viewon the content provider device. The content provider may view and editcontent data using the content provider front end. After editing thecontent data, the content data may then be saved to the content database114 for subsequent communication to a user device.

The content server 112 includes logic and data operative to formatcontent data and other information for communication to the user device.For example, the content data may be formatted to a content item whichmay be included in a stream of content items and advertisement itemsprovided to a user device. The formatted content items are specified byappearance, size, shape, text formatting, graphics formatting andincluded information, which may all be standardized to provide aconsistent look for all content items in the stream. In someembodiments, the content items have an associate bid amount which may beused for ranking or positioning the content items in a stream of itemspresented to a user device. In other embodiments, the content items donot include a bid amount or the bid amount is not used for ranking thecontent items. Such content items may be considered to be non-revenuegenerating items. The content server 112 then provides the content itemsto other network devices such as the advertising server 108 and theranking engine 116.

The ranking engine 116 is in data communication with the ad server 108,the ad database 110, the content server 112 and the content database114. The ranking engine 118 is configured to identify items to beincluded in a stream of content items and advertising items to beprovided to a user device such as user device 124. The ranking engine118 may thus be configured to determine which advertising items andwhich content items are qualified to be included in the stream and toscore and to order respective advertising items and respective contentitems in the stream.

In one embodiment, the ranking engine 116 is configured to calculate aranking score for each of a plurality of advertising items using bidvalues retrieved from the ad database 110. The ranking engine 116 isfurther configured to calculate a ranking score for each of a pluralityof content items using bid values obtained from the content database114. The ranking engine 116 may use other information available from thead server 108, the ad database 110, the content server 112 and thecontent database 114 as well as the account database 104 whendetermining the ranking scores. Other embodiments and other detail ofexemplary operation of the online information system 100 including theranking engine will be described below.

The account server 102, the search engine 106, the ad server 108, thecontent server 112 and the ranking engine 114 may be implemented as anysuitable computing device. A computing device may be capable of sendingor receiving signals, such as via a wired or wireless network, or may becapable of processing or storing signals, such as in memory as physicalmemory states, and may, therefore, operate as a server. Thus, devicescapable of operating as a server may include, as examples, dedicatedrack-mounted servers, desktop computers, laptop computers, set topboxes, integrated devices combining various features, such as two ormore features of the foregoing devices, or the like.

Servers may vary widely in configuration or capabilities, but generallya server may include one or more central processing units and memory. Aserver may also include one or more mass storage devices, one or morepower supplies, one or more wired or wireless network interfaces, one ormore input/output interfaces, or one or more operating systems, such asWindows Server, Mac OS X, Unix, Linux, FreeBSD, or the like.

The account server 102, the search engine 106, the ad server 108, thecontent server 112 and the ranking engine 114 may be implemented asonline server systems or may be in communication with online serversystems. An online server system may include a device that includes aconfiguration to provide content via a network to another deviceincluding in response to received requests for page views. An onlineserver system may, for example, host a site, such as a social networkingsite, examples of which may include, without limitation, Flicker,Twitter, Facebook, LinkedIn, or a personal user site (such as a blog,vlog, online dating site, etc.). An online server system may also host avariety of other sites, including, but not limited to business sites,educational sites, dictionary sites, encyclopedia sites, wikis,financial sites, government sites, etc.

An online server system may further provide a variety of services thatinclude, but are not limited to, web services, third-party services,audio services, video services, email services, instant messaging (IM)services, SMS services, MMS services, FTP services, voice over IP (VOIP)services, calendaring services, photo services, or the like. Examples ofcontent may include text, images, audio, video, or the like, which maybe processed in the form of physical signals, such as electricalsignals, for example, or may be stored in memory, as physical states,for example. Examples of devices that may operate as an online serversystem include desktop computers, multiprocessor systems,microprocessor-type or programmable consumer electronics, etc. Theonline server system may not be under common ownership or control withthe ad server 108, the content server 112 or the ranking engine 116.

The network 120 may include any data communication network orcombination of networks. A network may couple devices so thatcommunications may be exchanged, such as between a server and a clientdevice or other types of devices, including between wireless devicescoupled via a wireless network, for example. A network may also includemass storage, such as network attached storage (NAS), a storage areanetwork (SAN), or other forms of computer or machine readable media, forexample. A network may include the Internet, one or more local areanetworks (LANs), one or more wide area networks (WANs), wire-line typeconnections, wireless type connections, or any combination thereof.Likewise, sub-networks, such as may employ differing architectures ormay be compliant or compatible with differing protocols, mayinteroperate within a larger network such as the network 120. Varioustypes of devices may, for example, be made available to provide aninteroperable capability for differing architectures or protocols. Asone illustrative example, a router may provide a link between otherwiseseparate and independent LANs. A communication link or channel mayinclude, for example, analog telephone lines, such as a twisted wirepair, a coaxial cable, full or fractional digital lines including T1,T2, T3, or T4 type lines, Integrated Services Digital Networks (ISDNs),Digital Subscriber Lines (DSLs), wireless links including satellitelinks, or other communication links or channels, such as may be known tothose skilled in the art. Furthermore, a computing device or otherrelated electronic devices may be remotely coupled to a network, such asvia a telephone line or link, for example.

The advertiser device 122 includes any data processing device which mayaccess the online information system 100 over the network 120. Theadvertiser device 122 is operative to interact over the network 120 withthe account server 102, the search engine 106, the ad server 108, theranking engine 116, content servers and other data processing systems.The advertiser device 122 may, for example, implement a web browser forviewing web pages and submitting user requests. The advertiser device122 may communicate data to the online information system 100, includingdata defining web pages and other information. The advertiser device 122may receive communications from the online information system 100,including data defining web pages and advertising creatives.

In some embodiments, content providers may access the online informationsystem 100 with content provider devices which are generally analogousto the advertiser devices in structure and function. The contentprovider devices provide access to content data in the content database114, for example.

The user device 124 includes any data processing device which may accessthe online information system 100 over the network 120. The user device124 is operative to interact over the network 120 with the search engine106. The user device 124 may, for example, implement a web browser forviewing web pages and submitting user requests. A user operating theuser device 124 may enter a search request and communicate the searchrequest to the online information system 100. The search request isprocessed by the search engine and search results are returned to theuser device 124. In other examples, a user of the user device 124 mayrequest data such as a page of information from the online informationprocessing system 100. The data instead may be provided in anotherenvironment such as a native mobile application, TV application, or anaudio application. The online information processing system 100 mayprovide the data or re-direct the browser to another web site. Inaddition, the ad server may select advertisements from the ad database110 and include data defining the advertisements in the provided data tothe user device 124.

The advertiser device 122 and the user device 124 operate as a clientdevice when accessing information on the online information system 100.A client device such as the advertiser device 122 and the user device124 may include a computing device capable of sending or receivingsignals, such as via a wired or a wireless network. A client device may,for example, include a desktop computer or a portable device, such as acellular telephone, a smart phone, a display pager, a radio frequency(RF) device, an infrared (IR) device, a Personal Digital Assistant(PDA), a handheld computer, a tablet computer, a laptop computer, a settop box, a wearable computer, an integrated device combining variousfeatures, such as features of the forgoing devices, or the like. In theexample of FIG. 1, both laptop computer 126 and smartphone 128 may beoperated as either an advertiser device or a user device.

A client device may vary in terms of capabilities or features. Claimedsubject matter is intended to cover a wide range of potentialvariations. For example, a cell phone may include a numeric keypad or adisplay of limited functionality, such as a monochrome liquid crystaldisplay (LCD) for displaying text. In contrast, however, as anotherexample, a web-enabled client device may include one or more physical orvirtual keyboards, mass storage, one or more accelerometers, one or moregyroscopes, global positioning system (GPS) or otherlocation-identifying type capability, or a display with a high degree offunctionality, such as a touch-sensitive color 2D or 3D display, forexample. A client device such as the advertiser device 122 and the userdevice 124 may include or may execute a variety of operating systems,including a personal computer operating system, such as a Windows, iOSor Linux, or a mobile operating system, such as iOS, Android, or WindowsMobile, or the like. A client device may include or may execute avariety of possible applications, such as a client software applicationenabling communication with other devices, such as communicating one ormore messages, such as via email, short message service (SMS), ormultimedia message service (MMS), including via a network, such as asocial network, including, for example, Facebook, LinkedIn, Twitter,Mich, or Google+, to provide only a few possible examples. A clientdevice may also include or execute an application to communicatecontent, such as, for example, textual content, multimedia content, orthe like. A client device may also include or execute an application toperform a variety of possible tasks, such as browsing, searching,playing various forms of content, including locally stored or streamedvideo, or games. The foregoing is provided to illustrate that claimedsubject matter is intended to include a wide range of possible featuresor capabilities.

FIG. 2 illustrates streams of content items and data items displayed onselected user devices. In FIG. 2, a display ad 202 is illustrated asdisplayed on a variety of displays including a mobile web device display204, a mobile application display 206 and a personal computer display208. The mobile web device display 204 may be shown on the displayscreen of a mobile handheld device such as a smartphone. The mobileapplication display 206 may be shown on the display screen of a portabledevice such as a tablet computer. The personal computer display 208 maybe displayed on the display screen of a personal computer (PC).

The display ad 202 is shown in FIG. 2 formatted for display on a userdevice but not as part of a stream to illustrate an example of thecontents of such a display ad. The display ad 202 includes text 212,graphic images 214 and a defined boundary 216. The display ad 202 isdeveloped by an advertiser for placement on a web page sent to a userdevice operated by a user. The display ad 202 may be placed in a widevariety of locations on the web page. However, the defined boundary 216and the shape of the display ad must be matched to a space available ona web page. If the space available has the wrong shape or size, thedisplay ad 202 may not be useable.

To overcome these requirements and limitations, the display ad 202 maybe reformatted or alternately formatted for inclusion in a stream ofcontent items and advertising items including a stream ad incorporatingcontents of the display ad 202.

In these examples, the display ad is shown as a part of streams 224 a,224 b, and 224 c. The streams 224 a, 224 b, 224 c include a sequence ofitems displayed, one item after another, for example, down a web pageviewed on the mobile web device display 204, the mobile applicationdisplay 206 and the personal computer display 208. The streams 224 a,224 b, 224 c may include any type of items. In the illustrated example,the streams 224 a, 224 b, 224 c includes content items and advertisingitems. For example, stream 224 a includes content items 226 b and 228 aalong with advertising item 222 a; stream 224 b includes content items226 b, 228 b, 230 b, 232 b, 234 b and advertising item 222 b; and stream224 c includes content items 226 c, 228 c, 230 c, 232 c and 234 c andadvertising item 222 c. Each of the streams 224 a, 224 b, 224 c mayinclude any number of content items and advertising items. In oneembodiment, the streams 224 a, 224 b, 224 c may be arranged to appear tothe user to be an endless sequence of items so that as a user of a userdevice on which one of the streams 224 a, 224 b, 224 c is displayedscrolls the display, a seemingly endless sequence of items appears inthe displayed stream.

The content items positioned in any of streams 224 a, 224 b, 224 c mayinclude news items, business-related items, sports-related items, etc.Further, in addition to textual or graphical content, the content itemsof any stream may include other data as well, such as audio and videodata or applications. Each content item may include text, graphics,other data, and a link to additional information. Clicking or otherwiseselecting the link re-directs the browser on the user's device to a webpage referred to as a landing page that contains the additionalinformation.

Stream ads like the advertising items 222 a, 222 b, and 222 c may beinserted into the stream of content, supplementing the sequence ofrelated items, providing a more seamless experience for end users.Similar to content items, the advertising items may include textual orgraphical content as well as other data such as audio and video data orapplications. Each advertising item 222 a, 222 b, and 222 c may includetext, graphics, other data, and a link to additional information.Clicking or otherwise selecting the link re-directs the browser on theuser's device to a web page referred to as a landing page.

While the exemplary streams 224 a, 224 b, 224 c are shown with a singlevisible advertising item 222 a, 222 b, 222 c, respectively, any numberof advertising items may be included in a stream of items.Conventionally, it has been known to position the advertising items atfixed locations. For example, in one conventional system, it was knownto position an advertising item at the third item in the stream,counting from the top, at the sixteenth item in the stream and at everythirteenth item in the stream thereafter. That is, in the conventionalsystem, advertisements are located in pre-defined slots in the stream.Slotting of the advertisements is the same for all users under allconditions. In this regard, advertisements and content items arecomplements in the stream. If a content item is not placed at adesignated slot in the stream, an advertisement is placed in that slot.

In accordance with one aspect of the illustrated embodiment, slotting ofadvertisements in a stream is made dynamic. Any slot in the stream issubject to competition between advertising items and content items. Ascore is determined for each respective item. The scores for theadvertising items and the content items are made commensurate so thatadvertising items and content items may be ranked against each other andthe ranking used to populate the stream. Techniques for ranking theadvertising items and content items are discussed in further detailbelow.

FIG. 3 is a flow diagram illustrating one embodiment a method forranking and displaying a stream of advertising items and content itemsin an online information system. The method of FIG. 3 may be performedby, for example, elements of the online information system 100 of FIG. 1including the account server 102, the search engine 106, the ad server108, the content server 112 and the ranking engine 114. In otherembodiments, other components may be involved in performing the methodof FIG. 3 and some of the steps illustrated for the method of FIG. 3 maybe omitted or reordered and different steps may be added or substituted.

The method begins at block 300. At block 302, the method waits in a loopfor receipt of a page view request. A page view request is a datacommunication received over a network such as the network 120 of FIG. 1.The data communication includes data specifying a web page to view. Forexample, the page view request may specify the uniform resource locator(URL) for an online provider such as Yahoo! including the URL yahoo.com.The requested web page is one which may be populated in full or in partby a stream including items of at least two different types. In theexamples shown here, the types of items included in the stream arecontent items and advertising items, generally as shown in the exampleembodiment of FIG. 2. However, in other embodiments, other types ofitems may be provided in a stream, and the types of items or categoriesof items may be selected according to any convenient or useful criteria.For example, instead of populating a stream with content items andadvertising items as shown in FIG. 2, a stream may be populated withsports-related items of content and news-related items of content. Inanother example, instead of just two types of items being scored andranked together such as content items and advertising items, more thantwo items may be scored and ranked, including content items, CPCadvertising items and CPM advertising items. The method illustrated inFIG. 3 may be extended to the widest variety of combinations.

After a page view request has been received, at block 304, advertisementitems and content items are qualified so that only qualified items arethe subject of further processing. Items selected for qualification arecontained, in one example, in the ad database 110 and the contentdatabase 114 of the online information system 100 of FIG. 1.

Qualification may be performed on any suitable basis using any suitableinputs. For example, advertising items and content items may bequalified based on identification information for a user from whom thepage view request is received. If the online information system alreadystores information about interests and preferences of the identifieduser, that information may be used to qualify advertising items andcontent items. Also, if advertisers have specified targetingconstraints, such as gender, age and geography, those constraints may beapplied to information known about the user to qualify advertising itemsand content items. Still further, if the page request includesinformation specifying a device type or platform of the user device,such as smartphone or tablet or desktop computer, that platforminformation may be used to qualify advertising items and content itemsfor further processing. Some content providers may limit the contentitems they will send to particular platforms or format the content itemsto a particular format based on the platform information. Similarly,some advertisers may direct particular advertisement items only todesktop computers or tablet computers. Once the content items andadvertising items have been qualified, processing continues to block306.

At block 306 a clickability score is computed for each advertising itemand a clickability score is computed for each content item. Clickabilityis a measure of how many more clicks a given advertising item or contentitem gets compared to the average advertising item or content item. Inone example, clickability is a function of number of clicks anadvertising item or content item receives for all users and number ofimpressions or views an ad receives for all users, and the click throughrate (CTR) for the advertising item or the content item for all users.Clickability is position independent.

Click through rate is defined as the ratio of clicks to impressions anadvertising item or content items receives. Click through rate may bedetermined dynamically using stored data such as statistical data aboutperformance of advertisements in the online information system. Forexample, each time a particular advertising item or a particular contentitem is displayed or viewed by an advertiser, a data item representingviews or impressions for that item is incremented. Similarly, each timea particular advertising item or a particular content item is clicked orotherwise selected by a user, a data item representing clicks or clickthroughs for that item is incremented. Data items may be stored in an addatabase, a content database or any other suitable storage device suchas the ad database 110 and the content database 114 of FIG. 1.Similarly, mathematical processing may be performed, for example, by thead server 108, the content server 112 or the ranking engine 116 of FIG.1.

In one example, click through rate CTR is defined as

${CTR} = \frac{\sum{C_{i,t}^{{- \gamma}\; t}}}{\sum{V_{i,t}^{{- \gamma}\; t}}}$

where C_(i,t) represents the number of click throughs an advertisingitem or a content item receives at a particular position i over time t.Time t represents a discrete increment of time, the width of which mayvary. Typically, each increment of t corresponds to seconds or minutes.The position i refers to the position in the stream with i=1representing the first slot, i=2 the second, etc. In one embodiment,V_(i,t) represents the number of views or impressions the advertisingitem or content item receives at a particular position i over time t. Inanother embodiment, V_(i,t) represents the sum of clicks and skips for aposition i at time t. A skip for position i may be counted whenever auser clicks on an item in a position below position i, or may beadjusted by a coefficient to register a fractional count. For example,if a user clicks on i=4, positions i=1, 2, 3 have skip countsincremented at the same time that position i=4 has its click countincremented. The exponential term with coefficient gamma (γ) specifies adecay rate, which is typically longer for ad items than for content. Theexponentials introduce time-dependence so that, if an event such as aclick or view occurred recently in the past, it is given more weightthan older events.

In an embodiment, the coefficient gamma (γ) may be computed or adjustedbased on periodic comparisons of the click through rate of a given itemj at position i and time t+1 against the reference click through rateand clickability for the same item j at the same position i in theimmediately preceding time increment—e.g., by plottingrefCTR(i)*clickability(j,t) against CTR(i, j, t+1). The refCTR andclickability functions are further described below. In anotherembodiment, gamma (y) is computed by summing and then minimizing theerror:

γ=ΣV _(i,t)[refCTR(i)*clickability(j,t)−CTR(i,j,t+1)]²

For computing click through rate, the following decomposition is assumedfor both advertising items and content items:

CTR(ad/content, user, position, configuration)=Clickability(ad/content,user)*refCTR(position, configuration)

Thus, click through rate CTR is specified for a particular advertisingitem or content item, particular user, particular position in the streamand particular device configuration. Examples of device configurationsinclude a handheld device, a tablet and a desktop computer. Otherconfigurations and technologies may be accommodated as well and may beused to characterize CTR or other user data. The relationship betweenclickability and click through rate is specified by a reference curve,refCTR, which varies with position of the content item or theadvertising item in the stream and user device configuration.Clickability thus explicitly represents a position-independent CTR.

As a corollary, refCTR represents the probability that a user will clickon a particular advertising item or a particular content item at aparticular position i, independent of any effect on click through rateof the desirability (or undesirability) of particular advertising orcontent items. This refCTR may be computed by running a random bucketshowing click through rates for random advertising items or contentitems across all positions, with the same proportion ofadvertising/content as in the main bucket.

The value of clickability for an advertising item or content item isdesired to be position independent. Clickability eliminates any effectof position in the stream of a content item or an advertising item andfocuses instead on the quality of the content item or the advertisingitem.

Turning back to step 306 of FIG. 3, a clickability score for an item iin the stream may therefore be computed using the following generalrelation:

${{Clickability}(i)} = \frac{\sum\limits_{i}{C_{it}^{{- \gamma}\; t}}}{\sum\limits_{i}{V_{it}^{{- \gamma}\; t}*{{RefCTR}(i)}}}$

In some embodiments, however, it may be desirable to estimate or measurethe clickability for an item specific to particular users or aparticular market segment, which may be given an arbitrary index labelj. In such embodiments, clickability may be computed based on a relationspecific to user or market segment j:

${{Clickability}\left( {i,j} \right)} = \frac{\sum\limits_{i}{{{Click}s}\left( {i,j} \right)}}{\sum\limits_{i}{{{Impressions}\left( {i,j} \right)}*{{RefCTR}\left( {i,j} \right)}}}$

Thus, in various embodiments, to determine clickability, one may sumclicks over all positions and divide by the product of views orimpressions at all positions and refCTR.

In some embodiments, personalization is then introduced as an extrafactor, which contributes to the overall click through rate for a givenitem, user/segment, position, and device configuration:

CTR(ad/content, user/segment, position, config)=refCTR(position,configuration)*Clickability(ad/content,user/segment)*Affinity(ad/content, user/segment)

For content items, the Affinity between user and content is estimatedfrom historical data using the number of clicks observed for this useron content items with similar content features (with similaritydetermined using algorithmic analyses of similarity to context or knowncontent taxonomies, for example), normalized by the number of observedclicks from the reference user segment. In one embodiment, this is doneusing a Naive Bayes approximation. Many months of user historical datamay be used for reliably estimating this affinity for content items.

For advertising items, the affinity between user and an advertisementmay be more difficult to estimate from historical data. The user buyintent with respect to an advertised product or service is probably notas long lived as the user's general interest in a content item such as anews story. Therefore, a shorter historical window may be used foradvertising items. The exponential variation using gamma (γ) isintroduced to adjust the time window for which the quantity is beingcomputed. The historical data for advertising items also tends to bemuch more sparse. In some embodiments, search history profiles, mail, orother application activities may be used to expand the pool of userbehaviors. Regardless of the source of data, once the data is collected,an affinity score is computed by rendering both user data and items in ahigh-dimensional vector space defined by recognized features within anexisting content network or taxonomy. Description of such renderings areprovided in co-pending U.S. patent application Ser. No. 13/839,169,entitled “Method and System for Multi-Phase Ranking for ContentPersonalization,” and Ser. No. 13/837,357, entitled “Method and Systemfor User Profiling Via Mapping Third Party Interests to a UniversalInterest Space,” both of which are incorporated by reference herein intheir entirety.

After clickability is computed for each advertising item and eachcontent item at block 306, at block 308, a satisfaction score iscomputer for each advertising item and each content item. Satisfactionmay also be referred to as post-click satisfaction and is defined bysome measure of user satisfaction after the user has interacted with theadvertising item or the content item. Clickability generally onlymeasures the propensity of a user to click on an advertising item or acontent item. Satisfaction assigns a numerical value to the user'slikelihood to return to the online provider or the marketplace based onthe user's overall experience. In one example, satisfaction may be setto a value between 0 and 1 where 1 indicates complete user satisfactionand willingness to return and 0 indicates complete user dissatisfactionand a lost user who is unlikely to ever return.

In some examples, satisfaction can be estimated for an advertising itemusing conversion data. However, such data is relatively sparse and maynot be reliably compared across advertising items. In another example,dwell time may serve as a measure of satisfaction. Dwell time is anindication of the amount of time a user views a clicked-on advertisementafter clicking on an advertising item views content after clicking acontent item. In some embodiments, where insufficient data makesestimation unreliability, satisfaction may be set to a constant valuesuch as 1.0 for content items, advertising items or both.

In other embodiments, an additional factor for popularity may beincluded in the CTR model:

CTR(ad/content, user/segment, position, config)=refCTR(position,configuration)*Clickability(ad/content,user/segment)*Affinity(ad/content, user/segment)*Popularity(ad/content,user/segment)

The popularity score reflects a measure of overall interest in aparticular item within a window of time. The popularity score may becomputed, for example, based on a simple ranking of the highest clickthrough rates for ads or content within a window of time. The ranking isthen normalized and adjusted by a coefficient reflecting the relativeimportance of Popularity to the overall CTR.

At block 310, a bid amount for each content item is computed. Foradvertising items, a bid amount is set by the advertiser and is stored,for example, in the ad database with the rest of the data defining thead. For content items and advertising items to be ranked together forinclusion in the stream, there must be an analog to the bid amount of anadvertising item for a content item. In some embodiments, contentproviders may provide a bid amount if they are participating in theunified marketplace. In other embodiments, though, no content bids mayexist. Until publishers and other content providers explicitly bid toposition their content items in the stream, an automatic method maygenerate bids that allow ranking of content items and advertising itemstogether. In still other embodiments, content bids may reflect actualamounts paid to content contributors who are partnered with an onlineprovider.

One example for determining content bids is shown here. For each user oruser-segment, Table 1 below may be constructed. Here user-segment can bebased on the targeting profile of the user (i.e. the combination of usersegments that the advertiser can bid for). That is, if data for a givenuser is not available, data for the table can be obtained at anaggregate level based on user segments.

TABLE 1 Transformed Historical Bid (f[b_(a)]) (values below for m =Quality-Score (q_(c)) Percentile Historical bid (b_(a)) 0.7) 0.05 (minvalue of q_(c) 0 $0.10 (min value of $0.3 allowed) bid allowed) 0.15 10$0.25 $0.5 . . . 0.87 90 $1.50 $1.76 1.0 (max value of q_(c)) 100 $100(max possible $100 bid)

Table 1 stores percentile values for content-quality score andhistorical bids on ads. These percentiles are not impression-weighted.Thus the percentile distribution of quality-scores is over the set ofunique content items just as the distribution of bids is over the set ofunique ads. The percentile score may be obtained by selecting allcontent items in the ad database which are qualified for the user, theranking the selected content items by quality score. The ranked,selected items are then segregated according to their tenth percentile,twentieth percentile, etc. Other techniques may be used as well. Thus,the table translates quality score for content into a bid for content.In this manner, the quality score and the bid data are independent forcontent items and advertising items. Quality score for contents and biddata for ads is paired or correlated based only on relative percentile.

In other embodiments of the unified marketplace, ads and content may bepaired, correlated, or synchronized in other ways. For example, ads andcontent may be bundled based on similar subject matter in order to fitto an estimated overall clickability for the bundle, which is calibratedto preserve the quality of end user experience.

The Quality Score entered in Table 1 in one embodiment is a function ofclickability and post-click satisfaction. One technique for calculatingQuality Score is described below in conjunction with FIG. 4. Thehistorical period of the data may be restricted to ‘d’ days (where'd′ isan external parameter). The right-most column of Table 1 carries atransformation of the historical bid. An exemplary transformationfunction for bid amounts that may be used is the following:

$\left( {{{ba}\left\lbrack \max \right\rbrack} - {{{ba}\left\lbrack \min \right\rbrack}*\left( \frac{{ba} - {{ba}\left\lbrack \min \right\rbrack}}{{{ba}\left\lbrack \max \right\rbrack} - {{ba}\left\lbrack \min \right\rbrack}} \right)^{m}}} \right.$

where m∈(0 . . . M) where M is an input parameter. Values m<1 giveadvantage to content items and values m>1 give advantage to advertisingitems. Typically m would have a value of approximately 1. Data in Table1 can be updated every day by the online provider.

After content-items have been selected for the stream, upon a page viewevent, step 310 of the exemplary method of FIG. 3 includes a lookup foreach content-item from the table by using the Quality Score q_(c) as akey to the table. If an entry for q_(c) is found, look up thecorresponding bid b_(a) from the third column. If q_(c) is not found,the find the immediately higher score with an entry in the table,designated q_(c) ^(h). Then find the bid in the table which correspondsto q_(c) ^(h), b_(a). The transformed historical bid can then becomputed as b_(c)=k*f[b_(a)], where k is an external parameter.

Thus, the bid for the content item is determined using the bid for anadvertising item. This technique is exemplary only, but it ensures thatbids automatically determined for content items are commensurate withbids for advertising items and that each content item will get a uniquebid. These benefits are important for ranking the content items forinclusion in the stream. Moreover, the bid for a content item isproportional to its quality score. Thus, only high-quality advertisingitems, with high quality scores, will rise to the top of the stream andat the same time, content items with low quality scores will notdisplace high-quality advertising items. Since the content bid isproportional to the quality score, it reflects both long-term user valueas well as immediate short term revenue. Still further the content bidis proportional to the monetary value of the user since it can be safelyassumed that advertisers' bids reflect the value of the user. Finally,the illustrated technique is adaptive. That is, as bids on advertisingitems increase, bids on content items will increase as well. Up to apoint, this will induce advertisers to bid more.

The external parameters k and m may be chosen in any suitable manner. Inone example, the parameters may be chosen so that most the time, contentitems get the top slots in the stream, advertising items areinterspersed with content items and not clustered together. Also, themarketplace should achieve some level of stability after an initial bootup phase. This implies advertisers should not have to constantlyincrease their bids on their advertising items to hold their slots. Oneexception to this rule is when the marketplace is growing and newadvertisers are entering the marketplace.

At block 312, a ranking score is computed for each advertising item andfor each content item. For content items, in one embodiment,

ranking-score_(c) =b _(c) *c _(c) *s _(c)

where b_(c) is the computed bid for the content item, c_(c) is theclickability for the content item and s_(c) is the satisfaction scorefor the content item. For advertising items, in one embodiment,

ranking-score=b _(a) *c _(a) *s _(a)

where b_(a) is the advertiser-specified bid for the advertising item,c_(a) is the clickability for the advertising item and s_(a) is thesatisfaction score for the advertising item.

After computing the ranking score at block 312, at block 314, theadvertising items and the content items are sorted using their computedranking scores. Because the ranking scores have been computed usingsimilar values, the ranking scores are commensurate and may be reliablybe interlaced. The result of the sorting is a blended slate ofadvertising items and content items.

Following the sorting step, the blended slate may be used to populatethe stream. However, in some embodiments, it may be preferred to processthe blended slate for diversity, block 316. Diversity is applied tocontent to prevent too many similar content items to be located neareach other. An example is news articles on the same topic that may beranked close to each other. Only content items are affected by diversityprocessing and it will cause some content items to drop in rank.

One example of a diversity algorithm includes, first, ordering thecontent items by ranking-score (b_(c)*c_(c)*s) so that position 1corresponds to the highest score. Second, comparing the content items inposition 2-N , where N is in the range 200-300, against the content itemin position 1. Third, if any item in positions 2-N matches any featuresof the item in position 1, then multiply the item's ranking-score by adiversity score, such as a value in the range [0-1] that depends onfeature type. Fourth, the content items in positions 2-N may be sortedusing the new ranking scores penalized with diversity against thecontent item in position 1. Fifth, steps two through four may berepeated for content items in positions 3-N in comparison with thecontent item in position 2. And sixth, step five of repeating for itemsin positions 3-N may be repeated, for example, a minimum of 20 times.

In addition to processing content items for diversity, particular rulesmay be applied to advertising items following the sorting step, as well.For example, in one embodiment, rules may be applied as guard rails,preventing advertising items from being positioned in slots 1 and 2 ofthe stream. If an advertising item is in either of the first twopositions following the sorting process, the advertising item may beautomatically moved to a lower position such as slot 3. In anotherembodiment, a rule is established to maintain at least 9 content itemsbetween each advertising item in the stream. If fewer than 9 contentitems occur between two advertising items, the lower-ranked advertisingitem may be automatically moved to a lower position. Other values may bechosen for these rules, other rules may be established as well.

At block 318, pricing is calculated for advertising items and contentitems. Pricing refers to the amount an account of an advertiserassociated with an advertising item is charged in response to selectionof the advertising item by a user, such as by a click through.Similarly, pricing for a content item refers to the amount an account ofa publisher or content provider associated with a content item ischarged in response to selection of the content item by a user, such asby a click through of that content item.

In one embodiment a rule known as Generalized Second Price (GSP) isadopted for block 318. Under this rule, if an advertising item wasclicked on or otherwise selected by a user, the advertiser associatedwith the advertising item will be charged an amount which is equal tothe minimum bid required to win that position. Specifically, let idenote the advertising item a under consideration, and let i+1 be thecontent item or advertising item in the position right below advertisingitem a. Then, the price that the advertiser associated with advertisingitem a will be charged, if clicked, will be the maximum of a specifiedreserve price for item i and the quotient of the product of bid,clickability, and satisfaction for the item at i+1 and the clickabilityand satisfaction for the item at i:

$p_{i} = {\max\left\lbrack {\left( \frac{b_{i + 1}*c_{i + 1}*s_{i + 1}}{c_{i}*s_{i}} \right),{reserve}} \right\rbrack}$

The price p_(i) is always smaller than the original bid b_(;) sinceb_(i+1)*ci₊₁*s_(i+1)<b_(i)*c_(i)*s_(i). The price p_(i) also has thedesirable property of no subsequent regret for the advertiser. That is,if the advertiser would have chosen a bid smaller than b_(i), theadvertiser would still win the same slot. This is true as long as theadvertiser's bid is larger than p_(i). If the bid amount b_(i) ischarged to the advertiser, after the fact the advertiser would regretnot bidding p_(i) plus one cent. If instead we charge the advertiserp_(i) to begin with, the incentive to out-guess the system is reducedand advertiser satisfaction is increased. The advertiser would stillneed to check whether a different slot would be a better fit, though.Periodic bid adjustments are recommended.

In some embodiments, only cost per click (CPC) bids are accepted foradvertising items and content items. In other embodiments, however, costper impression (CPM) bids may be accepted as well. When CPM bids areaccepted, the desired allocation of bids to slots is required to placeads with higher expected revenue in higher slots, where a CPC ad aplaced in slot k is assumed to have expected revenueb_(a)*c_(a)*refCTR(k). When all bids are CPC bids, this allocation goalis achieved by simply sorting the advertising items according tob_(a)*c_(a) since refCTR(k) is advertisement independent and thereforecan be omitted. This makes the computation of the allocation of bidsrelatively quick and straightforward.

On the other hand, the revenue of a CPM ad does not depend in any way onits clickability or on its slot. The advertiser pays its bid wheneverthe ad is being displayed. Simply sorting CPC and CPM bids according tobid value (or bid amount multiplied by clickability) will no longersatisfy the property of higher revenue bids getting a higher position inthe stream.

However, the following approach may be used to jointly rank the twotypes of advertising items in one pass over slots, still satisfying theabove revenue requirement, and without computational slow down. Thus,there does not have to be any technical limitation in allowing both CPMand CPC ads being offered.

The proposed algorithm is as follows.

Input:

1. sorted CPC slate (in order of bid x clickability), x1>x2> . . .>x_(a)

2. refCTRs (position-dependent, ad independent) a1>a2> . . .

3. sorted CPM slate (in order of bid) y>Y2> . . . >Ym

For each slot k (starting with the highest slot, 1),

Let X be the highest-ranked CPC advertising item that isn't assigned yet

Let Y be the highest-ranked CPM advertising item that isn't assigned yet

If X*a_(k)>Y, then assign X to slot k, else assign Y to slot k.

The pricing technique may be varied for inclusion of CPM ads, as well.For a CPM ad, price is set at (bid *clickability of the CPM advertisingitem or content item below OR CPM of ad below). For a CPC ad, price isset at (bid * clickability of the advertising item or content itembelow)/(clickability of the CPC advertising item) If the advertisingitem below is CPC, OR price is set to (bid/refCTR of advertising itembelow)/(clickability of the CPC advertising item) if ad below is CPM.

This technique satisfies the above ranking-by-revenue requirement. Therun time and latency are not affected, since sorting time is thedominant factor here and sorting is being done even for CPC-only bids.

Illustrated in FIG. 4 is an example process for displaying content in astreaming media feed according to a quality score. Also illustrated isan example process for determining the quality score. In one example, auser of an electronic device, such as a mobile device, may be viewingcontent, such as news articles, in a streaming media feed. The content(e.g., news articles) may be interlaced with advertisements in the feedas in the exemplary illustration of FIG. 2. For example, the user mayview a news article, and when or after the article is viewed, the usermay request a next article; however, prior to that next article, anadvertisement may appear in the feed. In general, the order in whichadvertisements and articles appear in the feed may be arbitrary or bedetermined by a factor, such as a quality score.

In one example, the ranking engine of 116 of the online informationsystem 100 implements a quality scoring system (QSS), the operation ofwhich is shown schematically in FIG. 4. A processor of the rankingengine 115 cooperates with data stored in, for example, the ad database110 and the content database 114, to perform the following dataprocessing operations of the QSS.

The QSS receives information, designated in FIG. 4 as present content402, such as an advertisement or an article, to be scored by the QSS.The present content 402 may be received from a content source via anetwork, such as the Internet. The QSS may include an interface, such asan optical or electrical transceiver, configured to receive datadefining the present content 402 from a streaming media feed or anyother type of online feed. The content source may be any source ofadvertisements or multi-media content, such as a network of servershosting content configured to feed online content.

Upon receiving the present content in the streaming media feed, forexample via the interface of the QSS, a processor of the QSScommunicatively coupled to the interface may determine or be configuredto determine a first probability at 404, which is a probability a userwill select to interact with the item of present content in thestreaming media feed. In one example, the item of present content may bean advertising item or a content item as described and illustratedherein, for example in conjunction with FIG. 2.

Regarding the determination of the first probability at 404, selectingto interact with the present advertisement or article may includeclicking an advertisement or article or clicking a hyperlink to anadvertisement or article. Furthermore, such selecting may include agesture made with respect to an advertisement or article or a gesturemade with respect to a hyperlink to an advertisement or article.

As illustrated, the first probability may be based on data 406, whichcorresponds to features of the item of present content correlating to auser's prior interactions and/or predilections with content similar tothe present content. The interactions and/or predilections may come froma user profile stored in a database, such as database 408. The userprofile, for example, may include parameters associated with the userwith respect to viewing of streaming media content, for example, and thedata 406 may be received by the processor from a database 408 of theQSS.

Upon receiving the present content, the processor may determine or beconfigured to determine a second probability 410, which is a probabilitythe user will select to interact with content, such as an advertisementor an article, in general, in the streaming media feed. Data 412, whichis used as input to determine the second probability 412, may bereceived by the processor from the database 408 of the QSS.

The processor may also determine or be configured to determine anaffinity score of the present content. The affinity score is arelationship between the first determined probability and the seconddetermined probability at 414. The relationship may be represented by aratio or another type of numerical score, for example. Also, theaffinity score may represent a correlation between a user and a presentcontent item based on features of the present content item matching userprofile parameters associated with the user.

The affinity score may be independent of a configuration of the presentcontent item in a streaming media feed. For example, the affinity scoremay be independent of a position of the present content item in astreaming media feed. In other words, the affinity score may not takeinto account the order in which the present content appears with respectto other content in the streaming media feed.

In one example of the feed, the present content item may be a firstcontent item, such as an advertisement, and a second content item may bean article, such as a news article. In such a case, the first and thesecond content items may include similar subject matter categoricallyand the determining of the affinity score may include determining anaffinity score of the second content item based on the affinity score ofthe first content item. In such an example, the similarity of thesubject matter categorically may be identified by matching metadataelements embedded in the first and the second content items. Thematching metadata elements may include matching categories of a contenttaxonomy and/or associations with a webpage of a series of webpagesprovided by a web content provider. In one example, the webpage of aseries of webpages provided by a web content provider may be an onlineencyclopedia or dictionary entry, such as a WIKIPEDIA entry.

In another example of the feed, the determination of the affinity scoremay include determination of an affinity score between a device, such asa mobile device, of the user and the present content. This affinityscore between the user's device and the present content may be based ontext in the present content and a present geographic location of themobile device, for example. This affinity score may also be based on anyother attribute of the mobile device, such as a telecommunicationsservice carrier associated with the device, and on any other attributeof the present content, such as a video element of the present content.

In this other example, the processor may determine or be configured todetermine the first probability according to a probability a user of thedevice will select to view the present content item, based on featuresof the present content item matching profile parameters associated withthe user's device and/or the user. Regarding the second probability, theprocessor may determine it according to a probability a user using thedevice will select to view a content item, in general, in the streamingmedia feed. Finally, the processor may determine or be configured todetermine the affinity score based on a relationship between the firstdetermined probability and the second determined probability.

In yet another example, the processor may determine or be configured todetermine the first or the second probability using a machine learningtechnique. For example, the processor may determine the first or thesecond probability using a boosted decision tree or another form ofartificial intelligence.

The processor may also identify or be configured to identify data 416corresponding to post-interaction satisfaction with prior content, suchas data regarding post-interaction satisfaction with prior viewedadvertisements or articles that match categorically the present content.The data 416 may then be derived into a post-interaction satisfactionscore. The categorical matching may include matching by categories of acontent taxonomy and/or associations with a webpage of a series ofwebpages provided by a web content provider such as a provider of anonline encyclopedia or dictionary. The data 416 may include dataassociated with posts regarding the present content or similar content,including social media posts. The data 416 may also include dataregarding mouse clicks on the content or similar content or on links tosuch content. Also, the data 416 may include data regarding views ofsuch content, lengths of viewing such content, registration orsubscriptions to view such content, amount of sharing of such content toother users, and linking to such content via the user's own content.

Based on the affinity score and the post-interaction satisfaction data416 or the post-interaction satisfaction score, the processor maydetermine or be configured to determine the quality score at 418. Forexample, the quality score may be determined by computing a product ofthe affinity score and the post-interaction satisfaction score.

Upon determining the quality score, the score may be used by theprocessor as a basis for displaying the present content and/orconfiguring the present content with respect to the feed, such as theorder the present content is displayed relevant to other content in thefeed. Also, the quality score may be displayed or used in generatingreports for administrators, for example.

From the foregoing, it can be seen that the present disclosure providestechniques for an online provider to control the location, number anddensity of steam ads in a stream of content viewable by a user on a webpage. The stream may be viewed as a unified marketplace where bothcontent items and advertising items compete for inclusion in the stream.Scoring and ranking techniques permit commensurate ranking of bothcontent items and advertisement items. Additional business rules forcontent items and advertising items may further control the relativelocation of content items and advertisement items in a stream.

The disclosed method and system may be implemented partly in a server, aclient device, a cloud computing environment, partially in a server andpartially in a client device, or a combination of the server, the cloudcomputing environment and the client device.

It is therefore intended that the foregoing detailed description beregarded as illustrative rather than limiting, and that it be understoodthat it is the following claims, including all equivalents, that areintended to define the spirit and scope of this disclosure.

1. A method for determining a quality score for a content item in astreaming media feed that includes content items and advertisements,comprising: identifying, by a positioning system, a present geographiclocation of a mobile device; determining, by a processor communicativelycoupled to the positioning system, an affinity score between the mobiledevice and a present content item in a streaming media feed, based ontext in the present content item and the present geographic location ofthe mobile device; identifying post-interaction satisfaction with aprior content item that matches categorically the present content item;and determining, by the processor, a quality score based on the affinityscore and the post-interaction satisfaction; and displaying the contentitems with the advertisements in the streaming media feed based on thedetermined quality score, wherein a higher quality score correspondswith a higher position in the streaming media feed.
 2. The method ofclaim 1, wherein the determining of the affinity score further includes:determining a first probability, which is a probability a user of themobile device will select to view the present content item, based onfeatures of the present content item matching user profile parametersassociated with the user; determining a second probability, which is aprobability the user will select to view a content item, in general, ina streaming media feed; and determining the affinity score based on arelationship between the first determined probability and the seconddetermined probability.
 3. The method of claim 2, wherein thedetermining the first or the second probability uses a machine learningtechnique.
 4. The method of claim 2, wherein the determining of thefirst or the second probability uses a boosted decision tree.
 5. Themethod of claim 2, wherein the determining the quality score based onthe affinity score and the post-interaction satisfaction furthercomprises: deriving a post-interaction satisfaction score based on theidentified post-interaction satisfaction; and determining the qualityscore by computing a product of the affinity score and thepost-interaction satisfaction score.
 6. The method of claim 1, whereinthe affinity score is independent of a position of the present contentitem in a streaming media feed.
 7. The method of claim 1, wherein theaffinity score is independent of a configuration of the present contentitem in a streaming media feed.
 8. The method of claim 2, wherein thedetermining the quality score is further based on a popularity score. 9.The method of claim 8, wherein the popularity score is based on a clickthrough rate for the present content item within a window of time. 10.The method of claim 1, wherein the present content item is a firstcontent item, wherein a second content item is an article, wherein thefirst and the second content items include similar subject mattercategorically, and wherein the instructions to determine the affinityscore include instructions to determine an affinity score of the secondcontent item based on the affinity score of the first content item. 11.The method of claim 10, wherein the similarity of the subject mattercategorically is identified by matching metadata elements embedded inthe first and the second content items, and wherein the matchingmetadata elements are categories of a content taxonomy.
 12. The methodof claim 10, wherein the similarity of the subject matter categoricallyis identified by matching metadata elements embedded in the first andthe second content items, and wherein the matching metadata elements areassociations with a webpage of a series of webpages provided by a webcontent provider.
 13. The method of claim 12, wherein the webpage is anonline encyclopedia or dictionary entry.
 14. A system for unifieddisplay of a plurality of revenue and non-revenue generating items, thesystem comprising: a first database comprising a plurality of revenuegenerating items and historical bid values generated for the pluralityof revenue generating items; a second database comprising a plurality ofnon-revenue generating items; and a ranking engine with at least oneprocessor configured to calculate a ranking score for each of theplurality of revenue generating items using the historical bid values ofthe first database, and configured to calculate a ranking score for eachof the plurality of non-revenue generating items, wherein the rankingscore for the non-revenue generating items correlates with the rankingscore for the revenue-generating items such that the display of therevenue generating items with the non-revenue generating items isordered based on respective ranking scores.
 15. The system of claim 14,further comprising: displaying the revenue generating items in astreaming media feed based on the determined ranking score, wherein ahigher ranking score corresponds with a higher position in the streamingmedia feed.
 16. The system of claim 15, wherein the instructions todetermine the ranking score are further based on an affinity score and apost-interaction satisfaction and further comprise: deriving apost-interaction satisfaction score based on an identifiedpost-interaction satisfaction; and determining the ranking score bycomputing a product of the affinity score and the post-interactionsatisfaction score.
 17. The system of claim 16, wherein the affinityscore is independent of a position in the streaming media feed.
 18. Thesystem of claim 16, wherein the affinity score determination furthercomprises: determining a first probability, which is a probability theuser will select to interact with a content item in a streaming mediafeed; determining a second probability, which is a probability the userwill select to interact with a content item, in general, in thestreaming media feed, wherein the relationship between the firstdetermined probability and the second determined probability is a ratio;and determining the affinity score based on a relationship between thefirst determined probability and the second determined probability. 19.A system for determining a quality score for an advertisement astreaming media feed, comprising: a processor; and memory, the memorycommunicatively coupled to the processor, the memory comprisinginstructions executable by the processor for: identifying, by apositioning system, a present geographic location of a mobile device;determining, by a processor communicatively coupled to the positioningsystem, an affinity score between the mobile device and a presentcontent item in a streaming media feed, based on text in the presentcontent item and the present geographic location of the mobile device;identifying post-interaction satisfaction with a prior content item thatmatches categorically the present content item; and determining, by theprocessor, a quality score based on the affinity score and thepost-interaction satisfaction; and displaying the content items with theadvertisements in the streaming media feed based on the determinedquality score, wherein a higher quality score corresponds with a higherposition in the streaming media feed.
 20. The system of claim 20,wherein the determining of the affinity score further includes:determining a first probability, which is a probability a user of themobile device will select to view the present content item, based onfeatures of the present content item matching user profile parametersassociated with the user; determining a second probability, which is aprobability the user will select to view a content item, in general, ina streaming media feed; and determining the affinity score based on arelationship between the first determined probability and the seconddetermined probability.